Method and architecture to support interaction between host computer and remote display devices wherein service-specific protocols for the remote devices are defined

ABSTRACT

A method and architecture for enabling interaction between a remote device and a host computer. A service provided by the remote device is discovered, and a description pertaining to the service is retrieved by the host computer. A network communication link is the established between the remote device and the host computer based on connection information provided by the description. Host-side and client-side software service modules are run on the host and remote devices to enable interaction between the devices using a service protocol that is specific to the service. Various service protocols are provided, including a display service protocol and an input service protocol. Using commands provided by each protocol, the host computer is enabled to control the service remotely by pushing data and appropriate commands to the remote device, whereupon these commands are processed by the client-side service module to perform service operations that employ the sent data.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Divisional of U.S. patent application Ser.No. 09/989,136 filed Nov. 20, 2001 now U.S. Pat. No. 7,299,304.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention concerns interaction with remote devices ingeneral, and a method and system for performing remote interaction withlow-resource devices, including display, playback and synchronization ofmultimedia content and receiving input from such devices.

2. Background Information

Today, there are several software tools that enable users to interactwith a host computer from a remote location through software productsrunning on both a remote computer and the host computer. These softwareproducts generally fall under the category of “remote control” software,and include such products as PC Anywhere, Carbon Copy, Windows TerminalServer, and RDP (remote desktop protocol) for Microsoft Windowsoperating system environments, and X-Windows for UNIX and LINUXenvironments.

As shown in FIG. 1, these software products typically include a set ofclient-side software components 10 running on a remote computer 12 thatinteract with host-side software components 14 running on a hostcomputer 16 via a communication link over a network 18. Notably, theclient-side software components require both an operating system (OS) 20and computer hardware resources components to enable the operatingsystem to run, such as a high speed CPU 22 and memory 24. As a result,remote computer 12 requires both hardware and software (i.e., the OS)components that are fairly expensive. Furthermore, much of the hardwareand software components are duplicative when considering that similarhardware and OS software is required for host computer 16.

Recent advancements have also lead to the availability of standalonedevices that are used for displaying and/or generating multimediacontent. For example, digital picture frames are used to display one ormore digital pictures that are stored on a memory media card; in orderto change the pictures, the memory media card needs to be changed.Another common stand-alone device is the set-top box, which enablesusers to receive various multimedia content via a television network(e.g., cable or satellite network), and many models further enable usersto access the Internet via a dial-up or cable link.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same becomesbetter understood by reference to the following detailed description,when taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a schematic diagram illustrating a conventional architecturethat enables a host computer to be accessed via a remote computer;

FIG. 2 is a schematic diagram of an exemplary architecture that enablesa host or “extended” PC to interact with various remote devices inaccordance with the present invention;

FIG. 3 is a schematic diagram illustrating exemplary host-side andclient-side software components to enable an extended PC to interactwith a remote device;

FIG. 4 is a flowchart illustrating operations that are performed toenable interaction between a host computer and a remote device;

FIG. 5 is a schematic diagram illustrating a plurality of Universal Plugand Play (UPnP)-enabled devices, embedded devices, services, and controlpoints for explaining a typical UPnP environment

FIG. 6 is a block schematic diagram illustrating UPnP device and serviceadvertisement and control point search mechanisms;

FIG. 7 comprises a set of three tables presenting tabular informationpertaining to discovery messages broadcast during a UPnP discovery step;

FIG. 8 is a block schematic diagram illustrating how descriptioninformation is retrieved during a UPnP description step;

FIG. 9 is a schematic block diagram illustrating an exemplaryarchitecture in accordance with the invention that enables a hostcomputer to display content on a remote device;

FIG. 10 is a schematic block diagram illustrating an exemplaryarchitecture in accordance with the invention that enables a hostcomputer to send audio content to a remote device and have the audiocontent played back by the remote device;

FIG. 11 is a schematic block diagram illustrating an exemplaryarchitecture in accordance with the invention that enables various inputdevices operatively coupled to a remote device to provide input to ahost computer;

FIG. 12 is a schematic block diagram illustrating an exemplaryarchitecture corresponding to a PC/CE adapter that enables an extendedPC to push display and audio content to devices connected to the PC/CEadapter; and

FIG. 13 is a schematic diagram illustrating a personal computer that maybe used for an extended PC in accordance with the invention.

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

A system and method for displaying interacting with remote devices viaan extended PC host is described in detail herein. In the followingdescription, numerous specific details are provided, such as exemplaryservice protocols and system architectures, to provide a thoroughunderstanding of embodiments of the invention. One skilled in therelevant art will recognize, however, that the invention can bepracticed without one or more of the specific details, or with othermethods, components, etc. In other instances, well-known structures oroperations are not shown or described in detail to avoid obscuringaspects of various embodiments of the invention.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearances of the phrases “in one embodiment” or “in an embodiment” invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments.

The present invention provides a mechanism that enables interactionbetween a low-cost networked device (i.e., the remote device) and an“extended” personal computer, wherein a majority of the software andhardware components used to enable the interaction are provided by theextended PC, and the remote device requires limited hardware thatsupports “lightweight” software service components. The extended PC,which is also referred to as a “host” PC herein, is termed “extended”because it extends the reach of a PC in a home environment.

As shown in FIG. 2, the present invention enables interaction between aplurality of remote devices, including digital picture frames 30,electronic game boards 32, digital network tables 34 and televisionadapters 36, and an “extended” personal computer (PC) 38 (also referredto as the “host” or “host computer”) via a network 40. In addition tothe devices shown herein, the remote devices may include any devicecapable of hosting a network connection, including information terminalsand kitchen management devices. Typically, network 40 may comprise awired or wireless network, such as networks based on the 802.11a,802.11b, HPNA, Ethernet, HomeRF, HomePlug, IEEE 1394 (Firewire), etc,standards, or hybrid networks that use two or more of these networkstandards. Generally, the interactions between a remote device and anextended PC are enabled by host-side software components 42 running onextended PC 38 and a set of software service components running on theremote device. Exemplary software services components are enabled, inpart, by Universal Plug and Play (UPnP) modules 44, and include soundservice modules 46, display service modules 48, input service modules 50and video service modules 52.

An exemplary software architecture that facilitates interaction with adigital network tablet is shown in FIG. 3. The architecture includeshost-side software components 54 that interact with client-sidecomponents 56. The host-side software components include a pictureapplication 58, a jukebox application 60, a UPnP device abstractionlayer 62, a UPnP module 64, a sound module 66, an input module 68, adisplay module 70, and a network interface module 72. The client-sidesoftware services include a UPnP module 40, an input service module 50,a sound service module 46, and display service module 48.

A flowchart corresponding to a general high-level use-case of theinvention is shown in FIG. 4. In order to enable uni-and bi-directionalinteractions with remote devices, there needs to be a mechanism toestablish network communication between the extended PC and each remotedevice. As explained below in further detail, this process begins in ablock 80 by allocating an IP address for the device, and then using adiscovery mechanism that enables devices connected to an appropriatenetwork to be “discovered” in a block 81. After the device has beendiscovered, description information pertaining to the capabilities ofthe device and services provided by the device are retrieved in a block82. Using connectivity data provided by this information, a networkcommunication link is established between the extended PC and the remotedevice in a block 83.

In general, each device will provide one or more services that may berequested to be performed by other devices on the network. Theseservices may typically include “remote display,” “remote audio,” “remoteinput,” “remote video,” and other similar services. Typically, eachservice will be designed to interact with a corresponding servicecomponent (e.g., software module) running on extended PC 38. Forexample, for each client-side software service module shown in FIG. 3,there is a corresponding host-side software component running onextended PC 38.

Each of the software components running on extended PC 38 comprise oneor more software modules that are specifically-tailored to interact witha corresponding remote device service using a particular serviceprotocol. Depending on the services supported by a given host,corresponding host-side modules and protocols will be selectively loadedand enabled for execution on extended PC 38 in a block 84. Interactionbetween the host and client devices will then be performed usingappropriate service protocols in a block 85.

Establishing Network Connection, Discovery and Retrieving Capabilitiesusing UPnP

In one embodiment, establishing network communication, discovery ofdevices, and retrieval of those devices capabilities and services isenabled through Universal Plug and Play. UPnP is an architecture forpervasive peer-to-peer network connectivity that enables networkcommunication between UPnP-enabled devices using zero-configuration,“invisible” networking. Through UPnP, a device can dynamically join anetwork, obtain an IP address, convey its capabilities, and learn aboutthe presence and capabilities of other devices sharing the network. UPnPleverages Internet components, including IP, TCP, UDP, HTTP, and XML.Like the Internet, contracts are based on wire protocols that aredeclarative, expressed in XML, and communicated via HTTP. Furthermore,UPnP networking is media independent, does not require device drivers,and can be implemented using any programming language on any operatingsystem.

The basic building blocks of UPnP networks are devices, services andcontrol points. As shown in FIG. 5, each UPnP-enabled device comprises acontainer of one or more services and optional nested (i.e., embedded)devices and services. For instance, a UPnP device 90 comprises a singledevice 92 that provides two services 94 and 96, while a UPnP device 98comprises a single device 100 that provides a service 102, and a UPnPdevice 104 that includes a root device 106 that provides a service 108and an embedded device 110 providing services 112 and 114. Differentcategories of UPnP devices will be associated with different sets ofservices and embedded devices. Information identifying a given device,its services, and any embedded devices and their services are providedby an XML description document that is hosted by that device.

The smallest unit of control in a UPnP network is a service. A serviceexposes actions and models its state with state variables. For instance,a clock service could be modeled as having a state variable,current_time, which defines the state of the clock, and two actions,set_time and get_time, which allow control of the service. Similar tothe device description, service information is part of an XML servicedescription standardized by the UPnP forum. A pointer (URL) to theseservice descriptions is contained within the device descriptiondocument.

A control point in a UPnP network is a controller capable of discoveringand controlling other devices. For example, as shown in FIG. 5, controlpoints include devices used (e.g., solely) to control other devices, asdepicted by a control point 114 (e.g., extended PC 38), as well ascontrollers within devices that may be both controlled by other devices,as depicted by a control point 116. After discovery, a control pointmay: retrieve the device description and get a list of associatedservices; retrieve service descriptions for interesting services; invokeactions to control the service; and/or subscribe to the service's eventsource.

Addressing

The foundation for UPnP networking is IP addressing, which comprises“step 0” in UPnP networking. In one embodiment, each device provides aDynamic Host Configuration Protocol (DHCP) client that searches for aDHCP server when the device is first connected to the network. If a DHCPserver is available (i.e., a managed network exists), the device willuse the IP address assigned to it by the DHCP server. If no DHCP serveris available (i.e., the network is unmanaged), the device must useautomatic IP addressing (Auto-IP) to obtain an address. In brief, AutoIP defines how a device intelligently chooses an IP address from a setof reserved addresses and is able to move easily between managed andunmanaged networks. If during the DHCP transaction, the device obtains adomain name, e.g., through a DNS server or via DNS forwarding, thedevice should use that name in subsequent network operations; otherwise,the device should use its IP address.

A device that supports AUTO-IP and is configured for dynamic addressassignment begins by requesting an IP address via DHCP by sending out aDHCPDISCOVER message. The amount of time this DHCP Client should listenfor DHCPOFFERS is implementation dependent. If a DHCPOFFER is receivedduring this time, the device will continue the process of dynamicaddress assignment. If no valid DHCPOFFERS are received, the device maythen auto-configure an IP address. To auto-configure an IP address usingAuto-IP, the device uses an implementation-dependent algorithm forchoosing an address in the 169.254/16 range. The first and last 256addresses in this range are reserved and should not be used. Theselected address should then be tested to determine if the address isalready in use. If the address is in use by another device, anotheraddress must be chosen and tested, up to an implementation-dependentnumber of retries. Preferably, the address selection should berandomized to avoid collision when multiple devices are attempting toallocate addresses.

Once a device has a valid IP address for the network, it can be locatedand referenced on that network through that address. There may besituations where an end user (e.g., a user of extended PC) needs tolocate and identify a device. In these situations, a friendly name forthe device is much easier for a human to use than an IP address.Moreover, names are much more static than IP addresses. Clients thatrefer to a device by name don't need to be modified when IP address of adevice changes. Mapping of the device's DNS name to its IP address couldbe manually entered into DNS database or dynamically configured, if sucha facility is provided by the extended PC. While computers and devicessupporting dynamic DNS updates can register their DNS records directlyin DNS, it is also possible to configure a DHCP server to register DNSrecords on behalf of these DHCP clients.

Discovery

Once an IP address is obtained, discovery (UPnP networking “step 1”) maybe performed. When a device is added to the network, the UPnP discoveryprotocol allows that device to advertise its services to control pointson the network, as shown in FIG. 6. Control points can also discoverservices. When a control point is added to the network, the UPnPdiscovery protocol allows that control point to search for devices ofinterest on the network. The fundamental exchange in both cases is adiscovery message containing a few, essential specifics about the deviceor one of its services, e.g., its type, identifier, and a pointer tomore detailed information.

The UPnP discovery protocol is based on the Simple Service DiscoveryProtocol (SSDP). SSDP defines how network services can be discovered onthe network. SDDP is built on HTTPU and HTTPMU (variants of HTTP definedto deliver messages on top of UDP/IP instead of TCP/IP) and definesmethods both for a control point to locate resources of interest on thenetwork, and for devices to announce their availability on the network.

When a device is added to the network, it multicasts discovery messagesto advertise its root device, any embedded devices, and its services.Each discovery message contains four major components:

-   -   1. A potential search target (e.g., device type), sent in an NT        header;    -   2. A composite identifier for the advertisement, sent in a USN        header;    -   3. A URL for more information about the device (or enclosing        device in the case of a service), sent in a LOCATION header; and    -   4. A duration for which the advertisement is valid, sent in a        CACHE-CONTROL header.

To advertise its capabilities, a device multicasts a number of discoverymessages. With reference to FIG. 6, a root device will multicast threediscovery messages 120 for the root device, two discovery messages 122for each embedded device, and one discovery message 124 for eachservice. The content of these messages are shown in FIG. 7.

When a device is added to the network, it must send a multicast requestwith method NOTIFY and ssdp:alive in the NTS header in the followingformat, wherein values in italics are placeholders for actual values:

NOTIFY * HTTP/1.1 HOST: 239.255.255.250:1900 CACHE-CONTROL: max-age =seconds until advertisement expires LOCATION: URL for UPnP descriptionfor root device NT: search target NTS: ssdp:alive SERVER: OS/versionUPnP/1.0 product/version USN: advertisement UUID

The line HOST: 239.255.255.250:1900 specifies the multicast channel andport reserved for SSDP by the Internet Assigned Numbers Authority. TheCACHE-CONTROL value specifies a duration during which the advertisementis valid.

When a device and its services are going to be removed from the network,the device should multicast a ssdp:byebye message corresponding to eachof the ssdp:alive messages it multicasted that have not already expired.The following format is used for each message, wherein placeholders foractual values are in italics:

NOTIFY * HTTP/1.1 HOST: 239.255.255.250:1900 NT: search target NTS:ssdp:byebye USN: advertisement UUID

When a control point is added to the network, the UPnP discoveryprotocol allows that control point to search for devices of interest onthe network. It does this by multicasting a search message 126 with apattern, or target, equal to a type or identifier for a device orservice, as shown in FIG. 6. Responses 128 from devices that respond tothe search message contain discovery messages essentially identical tothose advertised by newly connected devices, which are unicast.

The multicast request is made with method M-SEARCH in the followingformat, wherein placeholders for actual values are in italics:

M-SEARCH * HTTP/1.1 HOST: 239.255.255.250:1900 MAN: “ssdp:discover” MX:seconds to delay response ST: search target

The ST: value must be in one of the following formats:

ssdp:all Search for all devices and services. upnp:rootdevice Search forroot devices only. uuid:device-UUID Search for a particular device.Device UUID specified by UPnP vendor.urn:schemas-upnp-org:device:deviceType:v Search for any device of thistype. Device type and version defined by UPnP Forum working committee.urn:schemas-upnp-org:service:serviceType:v Search for any service ofthis type. Service type and version defined by UPnP Forum workingcommittee.

To be found, a device must send a response to the source IP address andport that sent the request to the multicast channel. Responses toM-SEARCH are intentionally parallel to advertisements, and as such,follow the same pattern as listed for NOTIFY with ssdp:alive discussedabove, except that the NT header is replaced with an ST header. Theresponse is sent in the following format, wherein placeholders foractual values are in italics:

HTTP/1.1 200 OK CACHE-CONTROL: max-age = seconds until advertisementexpires DATA: when response was generated EXT: LOCATION: URL for UPnPdescription for root device SERVER: OS/version UPnP/1.0 product/versionST: search target USN: advertisement UUID

If the ST header in the request was:

ssdp:all Respond 3 + 2d + k times for a root device with d embeddeddevices and s embedded services but only k distinct service types. Valuefor ST header must be the same as the NT header in NOTIFY messages withssdp:alive. Single URI. upnp:rootdevice Respond once for root device.Must be upnp:rootdevice. Single URI uuid:device-UUID Respond once foreach device, root or embedded. Must be uuid:device-UUID. Device UUIDspecified by UPnP vendor. Single URI.urn:schemas-upnp-org:device:deviceType:v Response once for each device,root or embedded. Must be in the form ofurn:schemas-upnp-org:device:deviceType:v. Device type and versiondefined by UPnP Forum working committee.urn:schemas-upnp-org:service:serviceType:v Response once for eachdevice, root or embedded. Must be in the form ofurn:schemas-upnp-org:service:serviceType:v. Service type and versiondefined by UPnP Forum working committee.Description

The next step (2) in UPnP networking is description. After a controlpoint has discovered a device, the control point still knows very littleabout the device. For the control point to learn more about the deviceand its capabilities, or to interact with the device, the control pointmust retrieve the device's description from the URL provided by thedevice in the discovery message. Devices may contain other, logicaldevices, as well as functional units, or services. The UPnP descriptionfor a device is expressed in XML and typically includes vendor-specific,manufacturer information like the model name and number, serial number,manufacturer name, URLs to vendor-specific Web sites, etc. Thedescription also includes a list of any embedded devices or services, aswell as URLs for control, eventing, and presentation. For each service,the description includes a list of the commands, or actions, the serviceresponds to, and parameters, or arguments, for each action; thedescription for a service also includes a list of variables; thesevariables model the state of the service at run time, and are describedin terms of their data type, range, and event characteristics.

The UPnP description for a device is partitioned into two, logicalparts: a device description describing the physical and logicalcontainers, and one or more service descriptions describing thecapabilities exposed by the device. A UPnP device description includesvendor-specific, manufacturer information like the model name andnumber, serial number, manufacturer name, URLs to vendor-specific Websites, etc. (details below). For each service included in the device,the device description lists the service type, name, a URL for a servicedescription, a URL for control, and a URL for eventing. A devicedescription also includes a description of all embedded devices and aURL for presentation of the aggregate.

Note that a single physical device may include multiple logical devices.Multiple logical devices can be modeled as a single root device withembedded devices (and services), as shown in FIG. 8, or as multiple rootdevices (perhaps with no embedded devices). In the former case, there isone UPnP device description for the root device, and that devicedescription contains a description for all embedded devices. In thelatter case, there are multiple UPnP device descriptions, one for eachroot device.

A UPnP device description is written in XML syntax and is usually basedon a standard UPnP Device Template. A UPnP service description includesa list of commands, or actions, the service responds to, and parameters,or arguments, for each action. A service description also includes alist of variables. These variables model the state of the service at runtime, and are described in terms of their data type, range, and eventcharacteristics. This section explains the description of actions,arguments, state variables, and the properties of those variables.

Like a UPnP device description, a UPnP service description is written inXML syntax and is usually based on a standard UPnP Service Template.UPnP vendors can differentiate their devices by extending services,including additional UPnP services, or embedding additional devices.When a control point retrieves a particular device's description, theseadded features are exposed to the control point. The device and servicedescriptions authoritatively document the implementation of the device.

Retrieving a UPnP device description is relatively simple: the controlpoint issues an HTTP GET request on the URL in the discovery message,and the device returns the device description. Retrieving a UPnP servicedescription is a similar process that uses a URL within the devicedescription. The protocol stack, method, headers, and body for theresponse and request are explained in detail below.

As long as the discovery advertisements from a device have not expired,a control point may assume that the device and its services areavailable. The device and service descriptions may be retrieved at anypoint since the device and service descriptions are static as long asthe device and its services are available. If a device cancels itsadvertisements, a control point should assume the device and itsservices are no longer available.

The UPnP description for a device contains several pieces ofvendor-specific information, definitions of all embedded devices, URLfor presentation of the device, and listings for all services, includingURLs for control and eventing. In addition to defining non-standarddevices, UPnP vendors may add embedded devices and services to standarddevices. To illustrate these, below is a listing with placeholders initalics for actual elements and values corresponding to a devicedescription.

<?xml version=“1.0”?> <root xmlns=“urn:schemas-upnp-org:device-1-0”><specVersion> <minor>1</major> <minor>0</minor> </specVersion><URLBase>base URL for all relative URLs</URLBase> <device><deviceType>urn:schemas-upnp-org:device:deviceType:v</deviceType><friendlyName>short user-friendly title</freindlyName><manufacturer>manufacturer name</manufacturer> <manufacturerURL>URL tomanufacturer site</manufacturerURL> <modelDescription>long user-friendlytitle</modelDescription> <modelName>model name</modelName><modelNumber>model number</modelNumber> <modelURL>URL to modelsite</modelURL> <serialNumber>manufacturer's serialnumber</serialNumber> <UDN>uuid:UUID</UDN> <UPC>Universal ProductCode</UPC> <iconList> <icon> <mimetype>image/format</mimetype><width>horizontal pixels</width> <height>vertical pixels</height><depth>color depth</depth> <url>URL to icon</url> </icon> XML to declareother icons, if any, go here </iconList> <serviceList> <service><serviceType>urn:schemas-upnp-org:service: serviceType:v</serviceType><serviceId>urn:upnp-org:serviceId:serviceId</serviceId> <SCPDURL>URL forcontrol</controlURL> <controlURL>URL for eventing</eventSubURL></service> Declarations for other services defined by a UPnP Forumworking committee (if any) go here Declarations for other services addedby UPnP vendor (if any) go here </serviceList> <deviceList> Descriptionof embedded devices defined by a UPnP Forum working committee (if any)go here Description of embedded devices added by UPnP vendor (if any) gohere </deviceList> <presentationURL>URL forpresentation</presentationURL> </device> </root>

The UPnP description for a service defines actions and their arguments,and state variables and their data type, range, and eventcharacteristics. Each service may have zero or more actions, while eachaction may have zero or more arguments. Any combination of thesearguments may be input or output parameters. If an action has one ormore output arguments, one of these arguments may be marked as a returnvalue. Each argument should correspond to a state variable. Below is alisting with placeholders in italics for actual elements and valuescorresponding to a service description.

<?xml version=“1.0”?> <scpd xmlns=“urn:schemas-upnp-org:service-1.0”><specVersion> <major>1</major> <minor>0</minor> </specVersion><actionList> <action> <name>actionName</name> <argumentList> <argument><name>formalParameterName</name> <direction>in xor out</direction><retval /> <relatedStateVariable>stateVariableName</relatedStateVariable> </argument> Declarations for other argumentsdefined by UPnP Forum working committee (if any) go here <argumentList></action> Declarations for other arguments defined by UPnP Forum workingcommittee (if any) go here Declarations for other actions added by UPnPvendor (if any) go here </actionList> <serviceStateTable> <stateVariablesendEvents=“yes”> <name>variableName</name> <dataType>variable datatype</dataType> <defaultValue>default value</defaultValue><allowedValueList> <allowedValue>enumerated value</allowed value> Otherallowed values defined by UPnP Forum working committee (if any) go here</allowedValueList> <stateVariable> <stateVariable sendEvents=“yes”><name>variableName</name> <dataType>variable data type</dataType><defaultValue>default value</defaultValue> <allowedValueRange><minimum>minimum value</minimum> <maximum>maximum value</maximum><step>increment value</step> </allowedValueRange> </stateVariable>Declaration for other state variables by UPnP Forum working committee(if any) go here Declarations for other state variables added by UPnPvendor (if any) go here </serviceStateTable> </scpd>

Further details of how to implement addressing, discovery anddescription under UPnP are available through well-publicizeddocumentation, such as that available at the UPnP forum workingcommittee web site (www.upnp.org) and Microsoft's UPnP web site(www.upnp.com).

At this point, the present invention departs from the standardtechniques for controlling devices under UPnP, wherein interactionsbetween devices are defined by generalized UPnP control and eventingprotocols. Instead, the invention defines service-specific protocols toenable interaction between the extended PC and one or more remotedevices that host the services used. In general, these services mayinclude any service provided by a remote device, such as remote display,remote audio, remote input, and remote video; however, the invention isparticularly advantageous when considering that in a typicalimplementation the majority of the hardware and software resources arehosted by the extended PC while the resources required for each remotedevice are typically “lightweight” in comparison.

Lightweight Remote Display Service

In accordance with one aspect of the invention, a protocol is providedthat enables an extended PC to display content on a remote device usinga “push” model, wherein the extended PC pushes data and controlinformation to the remote device using a predefined set of commands, andthe data is formatted to meet display capabilities advertised by theremote device via the discovery mechanism discussed above. The serviceprovides a simple TCP connection port on which the extended PC sends astream of “display this picture at position X, Y” type commands

An exemplary set of protocol commands corresponding to a lightweightdisplay service are shown below, with accompanying descriptions.

Reset(int ProtocolVersion):

This primitive is used by the extended PC to reset the display on theremote device. The display must clear itself and await instructions.This primitive also clears any state the remote device may have. Theprotocol is defined with minimal state, but reset is defined such as tofuture proof the device. If future protocol versions are defined, thiscommand can be used to set which protocol version is going to be used bythe extended PC.

Flush(int TimeSinceLastSync)

Some display devices may support double buffering, where the devicewrites images to a hidden buffer and only displays the hidden buffer tothe screen when asked to do so. Double buffering results in less displayflicker. The “TimeSinceLastSync” parameter specifies the number ofrefresh cycles to wait before issuing the Flush command.

For example, suppose an NTSC TV adapter has double buffer support and itkeeps track of the screen refresh (about 30 times per second). Thedevice draws display data into its hidden buffer. When it receives aFlush(0) command, it displays the hidden buffer to the screen at thenext screen refresh. If it is desired to animate a GIF at 15 frames perseconds, the extended PC will need to send an animation frame to theremote device followed by a Flush(2) command. This will instruct thedevice to wait 2 refresh cycles before executing the Flush command.

FlushFailed( )

This command allows the PC to kept track of the quality of the animatedoutput and adjust it accordingly. If an animation frame is not receivedin time due to network and device loads, another mechanism comes intoplay to allow a PC to adaptively recover: If more than 2 refresh cyclesoccur before the reception of the Flush(2) command, a FlushFailed( )command is returned to the PC. This allows the PC to adapt the framerate and animation size to better fit the current network load andability of the device.

DrawFillBox(x1, y1, x2, y2, Color)

Draw a fill box of the given color defined by the corner point (x1,y2)and (x2,y2).

DrawImage(x, y, image)

Draw an image at the (x,y) location. Here, the image can be in manydifferent formats. When the PC discovers the remote device on thenetwork, it will retrieve the screen size, color depth and supportedimage formats for the device's and/or the device service's descriptioninformation. Information such as image width, height, and color are allencoded within the image format. The extended PC may use specialfeatures of some image types, such as alpha-blending and transparency,provided these capabilities are supported by the display device.

Allocate(Area Name, x, y, width, height)

This command may be used in cases where a remote device supportsstreaming video as a separate service. This command allows the PC toplace the video in a portion of the screen. Generally, the PC will setup the position of the video window using a configuration service orprotocol, and start streaming the video via a separate streaming servicesuch as UPnP/AV as defined by the UPnP forum.

Repaint( )

On occasion a device may lose the parts of the entire display content.To recover, the device can ask the extended PC to resend a completeupdate of the screen via the Repaint command.

Move(x1, y1, x2, y2, Width, Height)

This command moves a block from a portion of the frame buffer identifiedby (X1, Y1, Width, Height) to another portion of the frame bufferidentified by (X2, Y2, Width, Height). When using this command, devicesmay or may not support overlapping areas. If a device does not supportthis command at all, Drawimage commands are used instead.

In the most basic implementation, only Reset( ) and DrawImage( ) arerequired commands, while the remaining commands are optional. Perfectlysynchronized animations can be played on a display device that supportsthe Flush( ) and FlushFailed( ) commands, since the device animates theframes based on its own clock. If these commands are not available, theextended PC can still push data and commands to perform an animation,but the quality of the animation is subject to network conditions.

An exemplary software/hardware architecture that enables an extended PC38 to drive a display on a remote display device 128 is shown in FIG. 9.As discussed above, the majority of the hardware and software resourcesused to implement the lightweight remote display resides on the extendedPC. These include various software components 130, including a pictureapplication 58, a UPnP device abstraction layer 62, a UPnP module 64, adisplay module 70, and a network module 72. Client-side softwarecomponents 132, depicted as a UPnP service module 40 and a displayservice module 48, enable network communication and remote display andsynchronization to be performed by the remote display device in responseto data and commands sent over a network communications link 134 via anetwork interface 136. In general, network interface 136 may comprise anetwork interface that supports one or more of the network interfaceprotocols discussed above. Software components 132 will typically bestored as sets of machine instructions in a memory 140 that are executedby a controller 142 to perform the functions of each service. The remotedisplay device also includes a display buffer 144 that stores displaydata that is read by a display driver 146 to drive a display 148.

Generally, controller 142 may comprise any component enabled to performthe lightweight remote display functions discussed above throughexecution of software components 132. For example, such componentsinclude various types of microprocessors and microcontrollers.Furthermore, memory 140 will typically comprise some form ofnon-volatile memory, such as ROM, EPROM, Flash memory, etc. In oneembodiment, the operations performed by two or more of the hardwarecomponents may be provided by an Application Specific Integrated Circuit(ASIC) that is specifically designed to support the services provided bythe remote display device.

As shown in FIG. 9, remote display device 128 comprises an integrateddevice that includes both hardware and software components, as well asdisplay 148. Optionally, the hardware and software components may beemployed in a separate display adapter 150 that drives a display. Forexample, display 148 may comprise a television monitor 152 that receivesinput from a display adapter. Optionally, display 148 may represent adigital picture frame 154 that is driven by a display adapter.

In most instances, the remote display device will be provided with datacorresponding to a bitmap display. This enables the hardware on thedisplay device to be minimized. However, since bitmaps take a largeamount of memory and provide an inefficient means for storing imagecontent, most image and video data are stored in some compressed format,such as JPEG and GIF for still images, and MPEG for video images. As aresult, when using remote display devices that do not provide built-indecompression capabilities (either through built-in capabilities orprovided via client-side software (e.g., display module 48), it will benecessary to convert images from the compressed format into a bitmapformat prior to sending the image data to the remote display device.Optionally, if the remote device supports image decompression (eitherthrough built-in hardware or the client side software), image data maybe sent in a compressed format and decompressed at the remote displaydevice. Compression, decompression, and sending these data are beperformed on extended PC 38 through execution of display module 70.

Remote Audio

In addition to driving still image and video displays, the presentinvention also supports remote audio capabilities. Typically, the UPnPdescription of an audio service will provide information pertaining tothat device's audio capabilities, such as number of channels (i.e.,speakers), playback rate, quality (e.g., number of bits), and supportedsound format (e.g., PCM), voice input, etc.

An exemplary architecture to enable an extended PC 38 to drive a remoteaudio device 156 and/or receive audio input from the remote audio deviceis shown in FIG. 10. A number of software and hardware components usedin this architecture are substantially similar to those discussed abovewith reference to FIG. 9; these components share the same referencenumbers in both Figures. A set of software components 158 are running onextended PC 38, including a jukebox application 60 and a sound module 66in addition to UPnP device abstraction layer 62, UPnP module 64 andnetwork module 72 discussed above. Remote audio device 156 includes anaudio adapter 160 that includes software 162 stored in memory 140 thatincludes a sound service module 46. In addition to network interface 136and controller 142, the remote audio device also includes an audiodriver 164 that reads audio data stored in an audio buffer 166, andconverts the audio data into an amplified audio signal that drivesspeakers 168.

In one embodiment, sound input services are additionally provided byremote audio device 156. In this instance, a user may provide verbalinput via a microphone 165, which produces an analog audio signal thatis received by an audio input block 167 and converted into audio datahaving a digital format. Optionally, microphone 165 may directly convertverbal input into the digitized audio data. In one embodiment, audioinput block 167 will pass the digitized audio data to audio buffer 166,wherein it will be read by controller 142 and sent to extended PC 38 vianetwork interface 136. Optionally, audio input block 167 may directlypass the digitized audio data to controller 142.

Typically, the user's verbal input will correspond to a set of commandwords (or a command language with a predefined syntax) that are used asverbal commands to control actions performed by extended PC 38.Accordingly, sound module 66 will include voice recognition softwarethat will be used to interpret the audio data it receives to determinewhat command(s) the verbal input corresponds to.

In one embodiment, a TCP socket is opened to the remote audio device(via a port number provided by the service's UPnP description). Both thedevice and the extended PC TCP sockets can be optionally configured tobuffer only a small amount of data (depending on desired sound quality,network latency, and/or audio latency). The device simply reads PCMsound from the TCP socket and fills its audio buffer. If the remotesound device cannot perform these operations, it notifies the extendedPC that there is a problem. In one embodiment, sound module 66 includesone or more functions that enable the extended PC to adapt the soundquality (defined by its outgoing audio data stream) to meet a remotesound device's playback capabilities in consideration of initial and/orchanged network conditions. In one embodiment, the IETF defined RTPprotocol (RTP: A Transport Protocol for Real-Time Applications, RFC1889,January 1996) is used to carry PCM encoded audio in real time.

Lightweight Remote Input Service

Devices that may implement the invention can be built using varioustypes of input, from no input at all to pen-input, keyboard, mouse, andbutton input. Each type of input will generally be handled by arespective input service, wherein the existence and capabilities of eachinput service will be described in that service's UPnP descriptioninformation. This information may include type of input device, commandssupported (buttons, positions (e.g., pen, mouse position in XYcoordinates), clicks, z-axis position), etc.).

Some devices, such as infrared remote controls, might require coderunning on the extended PC to interpret the time codes it receives fromthe remote control and convert them into button commands. In thisinstance, the corresponding UPnP service might also be used to advertisethe presence of a custom input device and a URL link where the matchingPC software required to support the device is located so that it can bedownloaded to the extended PC over the Internet. Once downloaded, theextended PC can run the corresponding software component.

A basic set of primitives is supported by this lightweight remote inputservice. In one embodiment, these primatives include:

KeyDown(int keydata) KeyPress(int keydata) KeyUp(int keydata)

The three keyboard primitives match the well-known keyboard events usedis almost all operating systems. KeyDown and KeyUp occur whenever a keyis pushed or released, this generally includes all keys includingcontrol keys such as shift, ctrl, alt and function keys. The KeyPressevent contains the resulting interpretation of the keystrokes such assmall and large capitalization and keyboard repetitions.

MouseDown(int x, int y, int buttondata) MouseMove(int x, int y, intbuttondata) MouseUp(int x, int y, int buttondata)

The three pointer primitives match the well-known keyboard events usedis almost all operating systems. These primitives are not limited tomouse movements, any other pointing device such as trackballs, movementkeys, pen input, touch sensitive screen and magnetic screens can alsouse them. During input service discovery, the extended PC is informed ofthe maximum values of x and y. If the device has a display, thesepointer primitives may have a different resolution that may not matchthe x and y of the display. In such instances, software residing on theextended PC may be used to scale the pointer position to the display.

RemoteControlKey(int buttondata)

The framework also supports a remote control primitive to send toapplications button information that is not generally associated with acomputer keyboard. These buttons include various buttons that arecommonly found on remote control devices used to control audio and videoequipment, such as VCRs, DVD players, CD players, jukeboxes, etc.Typically, these buttons will include: Play, Pause, Forward, Reverse,Next Track, VCR, DVD, CD, etc. In some frameworks, the designer may optto not use this primitive and map these remote control keys to standardcomputer keyboard events via custom software running on the remotedevice and/or the extended PC.

Custom(int channel_number, byte[ ] custom_data)

In general, all keyboard, mouse and possibly remote control primitiveswill be dispatched to applications as-is. For some devices, dataprocessing will need to be performed before a keyboard or mouse eventcan be generated. In order to lower the cost of the device, it will bepreferable that device use this custom primitive to send raw input datato the extended PC for interpretation. The extended PC then sends theraw data into the appropriate code module for interpretation; the codemodule will generally return mouse or keyboard primitives that can thanby dispatched to the applications. In accordance with this primitive,each channel number can contain completely different raw data that maybe sent to a different respective code module for interpretation.

In general, a remote device does not have to use all of theseprimitives, but rather can use all or a subset of them. Some devices mayopt to use only custom data as input. An XML file will describe how theinput service behaves, what primitive, are used, data identifying thebounds of mouse coordinates, whether custom channels exist and where toobtain the code modules to interpret the raw input data. Preferably, theXML file will be written by the manufacturer of the device and validatedusing an XML-Schema file (XSD file). The Extended PC can also use theXSD file to validate the devices XML file before it is parsed.

Preferably, the code modules that are loaded and run by the extended PCshould be authenticated, especially in instances in which code modulesare downloaded from an Internet site. For example, authenticationtechniques such as digital signatures may be used to authenticate thecode modules. Techniques for authentication are well-known in the art,and so no further details are provided herein. The code modules shouldalso be designed to provide system safety. For instance, custom inputcode modules should be “sand-boxed” such that they are prevented fromtampering with system files and other software on the extended PC.

Another instantiation of this remote input service would do a similaroperation in reverse. A code module on the extended PC could be loadedto generate a custom command to a device. For example, if a device hasan IR transmitter, the PC could load a code module that sends an arrayof IR pulse timings to the device to be used for re-transmissionpurposes.

An exemplary architecture illustrating host-side and client-sidesoftware and hardware components to enable an extended PC 38 to receiveinput from a remote device 170 is shown in FIG. 11. As before, host-sidesoftware components include a UPnP device abstraction layer 62, a UPnPmodule 64 and a network module 72. New components that support remotedevice input services include an input module 68, a custom input module172, an infra-red (IR) remote module 174, a touch display module 176,and a button map module 178.

In a typical implementation, input capabilities will be supported by aninput adapter 180 provided for the remote device. The input adapter maycomprise circuitry built into the remote device, or may comprise aseparate add-on component. As with the display and audio adaptersdiscussed above, the input adapter includes a network interface 136, acontroller 142, and memory 140. In this instance, memory 140 includessoftware 182 comprising a UPnP client-side module 40 and a client-sideinput module 50. The input adapter further includes an input signalprocessing block 184, which may be performed by hardware and/or softwarecomponents that are well-known in the art for processing signals frominput devices. Optionally, the input adapter may also include an inputbuffer 186.

Generally, input adapter 180 will be designed to receive input from oneor more input devices 188, which are exemplified by wired and wirelesskeyboards 190, wired and wireless pointing devices 192, and remotecontrol devices 194.

An exemplary architecture provided by a PC/CE adapter 200 thatimplements a plurality of the services discussed above is shown in FIG.12. In this instance, the PC/CE adapter includes various softwarecomponents that enable the adapter to interact with an extended PC 38,drive audio and video signals for a television (or television monitor)202, drive a digital picture frame 204, and receive input from an IRremote control 206. These software components include a UPnP module 40,a sound service module 46, a display service module 48, and an inputservice module 50. The sound and display service modules interact withappropriate hardware (not shown) to provide analog legacy audio andvideo signals 208 and 210 to a television 206. Also, digital audio andvideo signals may be sent using a 1394 link. Display service module 48also is used to provide bitmap data to digital picture frame 204 via aIEEE 1394 connection 212. The PC/CE adapter is accessed via an IEEE802.11a wireless connection 214 and provides a IEEE 1394 bridge 216 thatenables IEEE 1394 signals to be interfaced with IEEE 802.11a signals.

Exemplary Computer System for Practicing the Invention

With reference to FIG. 13, a generally conventional computer 300 isillustrated, which is suitable for use as the extended PC in connectionwith practicing the present invention, and may be used for runninghost-side software comprising one or more software modules thatimplement the various host-side functions of the invention discussedabove. Examples of computers that may be suitable for hosts as discussedabove include PC-class systems operating the Windows XP, 2000, or NToperating systems, Sun workstations operating the UNIX-based Solarisoperating system, and various computer architectures that implementLINUX operating systems. Computer 300 is also intended to encompassvarious server architectures, as well as computers having multipleprocessors.

Computer 300 includes a processor chassis 302 in which are mounted afloppy disk drive 304, a hard drive 306, a motherboard 308 populatedwith appropriate integrated circuits including memory 310 and one ormore processors (CPUs) 312, and a power supply (not shown), as aregenerally well known to those of ordinary skill in the art. It will beunderstood that hard drive 106 may comprise a single unit, or multiplehard drives, and may optionally reside outside of computer 300. Amonitor 314 is included for displaying graphics and text generated bysoftware programs and program modules that are run by the computer. Amouse 316 (or other pointing device) may be connected to a serial port(or to a bus port or USB port) on the rear of processor chassis 302, andsignals from mouse 316 are conveyed to the motherboard to control acursor on the display and to select text, menu options, and graphiccomponents displayed on monitor 316 by software programs and modulesexecuting on the computer. In addition, a keyboard 318 is coupled to themotherboard for user entry of text and commands that affect the runningof software programs executing on the computer. Computer 300 alsoincludes a network interface card 320 or built in network adapter forconnecting the computer to a computer network, such as a local areanetwork, wide area network, or the Internet.

Computer 300 may also optionally include a compact disk-read only memory(CD-ROM) drive 322 into which a CD-ROM disk may be inserted so thatexecutable files and data on the disk can be read for transfer into thememory and/or into storage on hard drive 306 of computer 300. Other massmemory storage devices such as an optical recorded medium or DVD drivemay be included. The machine instructions comprising the software thatcauses the CPU to implement the functions of the present invention thathave been discussed above will likely be distributed on floppy disks orCD-ROMs (or other memory media) and stored in the hard drive untilloaded into random access memory (RAM) for execution by the CPU.Optionally, all or a portion of the machine instructions may be loadedvia a computer network.

Although the present invention has been described in connection with apreferred form of practicing it and modifications thereto, those ofordinary skill in the art will understand that many other modificationscan be made to the invention within the scope of the claims that follow.Accordingly, it is not intended that the scope of the invention in anyway be limited by the above description, but instead be determinedentirely by reference to the claims that follow.

1. A method for displaying content on a remote display device,comprising: establishing a network communication link between the remotedisplay device and a host computer; determining display capabilities ofthe remote display device based on a capabilities advertisement receivedby the host computer; launching a host-side software module to run onthe host computer to enable interaction with the remote device via adisplay service-specific protocol that is specific to the displaycapabilities of the remote display device, wherein the displayservice-specific protocol is determined from the capabilitiesadvertisement; sending display data corresponding to the display contentfrom the host computer to the remote display device over the networkcommunication link, said display data having a format corresponding tothe display capabilities of the remote display device; sending displaycommands corresponding to the display service-specific protocolindicating how the display data is to be displayed on the remote displaydevice; displaying the display data on the remote display device inresponse to the display commands; and wherein the displayservice-specific protocol includes feedback primitives to enable theremote display to provide display feedback information to the hostcomputer.
 2. The method of claim 1, wherein displaying the display dataon the remote display device comprises displaying the display data on adigital picture frame.
 3. The method of claim 1, wherein displaying thedisplay on the remote display device comprises displaying the display ona television monitor via a display adapter.
 4. The method of claim 1,wherein establishing the network communication link comprises:connecting the remote display device to a network to which the hostcomputer is already connected; obtaining an IP address for the remotedisplay device; broadcasting information pertaining to at least oneservice provided by the remote display device that includes the IPaddress over the computer network; and establishing a networkcommunication link between the remote display device and the host of theremote display device that uses the IP address of the remote displaydevice and an IP address previously assigned to the host computer. 5.The method of claim 1, wherein the display service-specific protocolincludes display synchronization commands that are sent to the remotedisplay device to enable the display content to be refreshed inaccordance with a predetermined timing to produce synchronizedanimations.